Detecting Errors within a Corpus using Anomaly Detection

We present a method for automatically detecting errors in a manually marked corpus using anomaly detection. Anomaly detection is a method for determining which elements of a large data set do not conform to the whole. This method fits a probability distribution over the data and applies a statistical test to detect anomalous elements. In the corpus error detection problem, anomalous elements are typically marking errors. We present the results of applying this method to the tagged portion of the Penn Treebank corpus. 1 I n t r o d u c t i o n Manually marking corpora is a time consuming and expensive process. The process is subject to human error by the experts doing the marking. Unfortunately, many natural language processing methods are sensitive to these errors. In order to ensure accuracy in a corpus, typically several experts pass over the corpus to ensure consistency. For large corpora this can be a tremendous expense. In this paper, we propose a method for automatically detecting errors in a marked corpus using an anomaly detection technique. This technique detects anomalies or elements which do not fit in with the rest of the corpus. When applied to marked corpora, the anomalies tend to be errors in the markings of the corpus. To detect the anomalies, we first compute a probability distribution over the entire corpus. Then we apply a statistical test which identifies which elements are anomalies. In this case the anomalies are the elements with very low likelihood. These elements are marked as errors and are thrown out of the corpus. The model is recomputed on the remaining elements. At conclusion, we are left with two data sets: one the normal elements and the second the detected